A brief survey of techniques for learning distributed representations

One of the basic problems with distributed representations is coming up with the codes for the particular objects to be represented. A distributed representation that is useful for solving a particular problem is usually a re-representation of the raw input (possibly sensory data). It represents, in a salient manner, the semantic features of the input (i.e., features that are relevant to solving the problem). The raw inputs to a neural network often carry few or no clues, in terms of simple dot-product similarity of activity in the input units, as to which inputs should be treated as similar or identical, and which should be treated as different. For a feedforward neural network such as the one in Figure 1 to solve such a problem in a parsimonious fashion the inputs must be re-represented, in the hidden layers, in a manner such that semantically similar, but neurally dissimilar, inputs evoke neurally similar activation patterns in hidden layers. E.g., if input patterns A and B are neurally dissimilar (i.e., have low dotproduct similarity) but should produce the same output from the network, then one way for a network to do that is to have the weights entering the hidden layer transform the neurally dissimilar input layer patterns A and B into the neurally similar hidden-layer patterns A and B. The techniques used in the literature for developing distributed representations can be classified into five broad categories: (1) random; (2) hand constructed; (3) learned in a supervised neural network from examples of inputoutput mappings; and (4) learned in an unsupervised neural network from examples of typical inputs (with no outputs presented); and (5) derived mathematically from summary statistics of data. Unfortunately, there have not been any successful attempts to systematically learn, on a large scale, distributed representations for objects with complex structure.